Looking at refactoring rucio-task-manager reports (adding graphics from dashboard).
Looking at refactoring rucio-task-manager reports.
Refactored rucio-task-manager so IAM events are sent to database. Still need to refactor daemons probe and reports. Events should be continuous and not rewritten, and should use kubernetes service accounts.
Refactoring of sync scripts.
Looking at how to deploy internal loadbalancer for ingress again.
Q: Why do we have to load balance the ingress-nginx service, and not the service it's ingressing?
A: Reduces the number of load balancers required. Doesn't need one per service.
Looking at how to deploy internal loadbalancer for ingress.
Found that there are two separate ingress controllers, one community maintained (ingress-nginx) and one by nginx (nginx-ingress). The latter does not support blank hosts, so will need to migrate to the former.
More Rancher bits (upgrading k8s version etc.)1
Looking at how to deploy Octavia load balancers for a service. Trying to get it to use an existing IP (with ports opened), but looks to me like the openstack provider always tries to CREATE a floating ip, even if it already exists?
this causes an issue with permissions. Neutron doesn't allow anyone but admin to create an arbitrary IP.
Some PI16 discussions.
Looking at load balancer with a fixed ip.
Some meetings including CADC.
Looking at Rancher (integration with cloud controller manager e.g. load balancer deployment).
Node templates can provide access to OpenStack when bootstrapping, but if you want to provision resources in the cluster you have to separately define a cloud_provider that is "injected" into the cluster on init/update.
Moved grafana OIDC login from ESCAPE to Grafana with the help of Tom Dack at STFC.
Playing around with rucio-events dash. Some work on cluster migration plan.
Cleaning up rucio-task-manager and adding ability to aggregate transfer events.
Playing around with rucio-events dash. Some work on cluster migration plan.
Playing around with rucio-events dash.
Fixed issues with reaper (lingering x509 proxy). Tidied up 474/478 confluence notes to make it more of a recipe.
Reading about and testing rancher deployment.
Not reaping imperial, cnaf or spsrc_storm.
Some issue with CA certificates (which are mounted as volumes from secrets) in the reaper pod. Did a strack trace of the davix call with and without --capath . with it, it looks in /etc/grid-security/certificates , without it it doesn't
[root@daemons-reaper-696559dfd7-ncxns ~]# strace davix-ls -l -H "Authorization: Bearer $BEARER_TOKEN" https://gfe02.grid.hep.ph.ic.ac.uk:2880/pnfs/hep.ph.ic.ac.uk/data/escape/ska/ |& grep /etc/grid
[root@daemons-reaper-696559dfd7-ncxns ~]# strace davix-ls -l --capath /etc/grid-security/certificates -H "Authorization: Bearer $BEARER_TOKEN" https://gfe02.grid.hep.ph.ic.ac.uk:2880/pnfs/hep.ph.ic.ac.uk/data/escape/ska/ |& grep /etc/grid
execve("/usr/bin/davix-ls", ["davix-ls", "-l", "--capath", "/etc/grid-security/certificates", "-H", "Authorization: Bearer eyJraWQiOi"..., "https://gfe02.grid.hep.ph.ic.ac."...], 0x7ffe16b1f5a0 /* 76 vars */) = 0
stat("/etc/grid-security/certificates", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=18760, ...}) = 0
stat("/etc/grid-security/certificates/530f7122.0", {st_mode=S_IFREG|0644, st_size=1367, ...}) = 0
open("/etc/grid-security/certificates/530f7122.0", O_RDONLY) = 4
stat("/etc/grid-security/certificates/530f7122.1", 0x7ffd373deee0) = -1 ENOENT (No such file or directory)
stat("/etc/grid-security/certificates/7ed47087.0", {st_mode=S_IFREG|0644, st_size=1281, ...}) = 0
open("/etc/grid-security/certificates/7ed47087.0", O_RDONLY) = 4
stat("/etc/grid-security/certificates/7ed47087.1", 0x7ffd373deee0) = -1 ENOENT (No such file or directory)
by default gfal doesn't seem to (???):
[root@daemons-reaper-696559dfd7-ncxns ~]# strace gfal-ls https://gfe02.grid.hep.ph.ic.ac.uk:2880/pnfs/hep.ph.ic.ac.uk/data/escape/ska/ |& grep /etc/grid
open("/etc/grid-security/gsi.conf", O_RDONLY) = 3
stat("/etc/grid-security/hostcert.pem", 0x7ffe9f21c850) = -1 ENOENT (No such file or directory)
write(5, "/etc/grid-security/hostcert.pem/"..., 2823) = 2823Paused the python process in the reaper container (so it could actually process commands).
Looked at deploying Rancher with Rohini. Provisioned a new cluster with it.
Importing of rucio-analysis transfer tests to argocd. Added workaround for external helm chart values file.
Looked a bit at ClusterAPI. Also some Rucio housekeeping (fixing IMPERIAL certs etc.)